In many data storage systems, the use of an array of distributed storage has been well explored to improve performance and reliability of data storage. Through using an array of distributed storage, performance may be improved by, for example, (a) permitting data reads or writes for portions of a requested data object, file or chunk concurrently from multiple disks or (b) permitting reads to be returned from the disk with the lowest latency, if there are redundant versions of data (note that this may have an adverse effect on writes since a complete write operation is only as fast as the slowest disk to which it is written). Reliability may be increased if mirroring or parity is used to provide redundant copies of data (or the data that permits the recreation of lost data).
For example, RAID is a common technology used with arrays of data storage components to increase the performance and/or reliability of data storage on those components. RAID is abbreviation that stands for “Redundant Array of Inexpensive Disks.” A data storage system that implements RAID may consist of two or more storage components working together to serve up data. These components have historically been hard disks, but other types of data storage components may be used, including technology for SSD (solid state drives). Depending on requirements and circumstances, various RAID levels may be implemented, each optimized for a specific situation. While these naming conventions have not been standardized by an industry group or standards body, they do appear to have developed as a more or less reliable naming convention over time, although some entities may use their own naming conventions to describe similar technologies. Some of the RAID levels include RAID 0 (striping), RAID 1 (mirroring), RAID 5 (striping with parity), RAID 6 (striping with double parity), and RAID 10 (combination of mirroring and striping).
The software to perform the RAID-functionality and control the drives can either be located on a separate controller card (a hardware RAID controller) or it can simply be a driver. Some versions of Windows, such as Windows Server 2012 as well as Mac OS X, include software RAID functionality. Hardware RAID controllers cost more than pure software but they also offer better performance. RAID-systems can be based with a number of interfaces, including SCSI, IDE, SATA or FC (fibre channel.) There are systems that use SATA disks internally but have a FireWire or SCSI-interface for the host system. Sometimes disks in a storage system are defined as JBOD, which stands for ‘Just a Bunch Of Disks’. This means that those disks do not use a specific RAID level and act as stand-alone disks. This is often done for drives that contain swap files or spooling data.
Since most modern distributed disk systems have been based on systems that resemble or have evolved from RAID methodologies, such systems arrange data depending on pre-determined data placement decision-making schemes that attempt to balance various key objectives. These include performance and reliability, but may also include availability and capacity. For example, RAID technologies assign data storage resources that distribute portions (e.g. a stripe or a chunk) of a discrete set of data, such as a file or object, across an array of independent disks. An associated file system maintains an index of the data locations, or otherwise manages the locations of the stored data. A decision-making scheme decides where to place each portion on each disk based on a set of static constraints, such static constraints being fixed at the time when the data is assigned to specific locations. When one of the devices fails on which a primary replicate is being used, the system finds the back-up copy and either uses that copy or writes a new copy (either copied from another copy that is a mirror or rebuilds the copy using parity) to the previously failed replica location. Load balancing may also be implemented by distributing some portion of replicates to the least loaded device in the array (which may support more than one of the key objectives discussed above). Each of these performance requirements may be considered a constraint that must be maintained by the system. For example, RAID 5 requires single parity for all data. RAID 0 requires at least 1 replica (or mirror) for all data. In other cases, ensuring that not all replicas exist in the same failure domain (e.g. different disks on the same server rack may be considered the same failure domain).
Many modern data storage systems have implemented improvements to be scalable and adaptive. Scalable means that storage resources, including specific tiers of data storage resources that may be required depending on the nature of at least some of the data stored in the system as well as the uses of that associated data, can be added to the data storage system over time as the requirements for the data begin to approach or exceed the available resources.
Moreover, recent developments in data storage have recognized that different data should be treated differently at different times; in other words adaptive. For example, some specific sets of data may be associated by an increased demand (or likelihood for demand) for access or writing/updating, or an increased likelihood of such demand, at identifiable or predictable times. Some data, or the processing thereof, may be associated with a significant value for very low latency, availability, reliability, or other performance-based metric (and such association may vary over time or in connection with some other characteristic, including but not limited to the identity or location of the data client). These examples may be considered high priority or “hot” data. Conversely, infrequently accessed data, or data that does not require specific performance guarantees, may be considered “cold” or low priority data. For example, high priority data may include information relating to trades on a public stock exchange requiring near instantaneous updating. Additionally, banking and shopping information may require near perfect locking from concurrent requests; in contrast, low priority data may include data used to install an operating system and which may be requested or changed only very infrequently, and even when it is changed, does not require low-latency. Modern storage systems have devised improvements for pushing high priority data with appropriately performing data storage resources (i.e. higher tiers). High tier performance for high priority data may be considered a constraint.
For all of the above reasons, as well as others, there is a requirement to be able to dynamically assign specific data resources, or classes of data resources, for specific reasons and specific uses, in order to meet a set of constraints associated with the storage of that data. Known systems have implemented techniques for distributing data within a data storage system based on an initial set of static data storage constraints, but techniques for optimizing compliance, particularly as data characteristics change and the data storage system changes, and as the data storage constraints themselves are changed (including adding new ones or removing old ones), have not been well addressed.
During the assignment of resources, particularly as circumstances relating to a given data storage system change over time, rapidly or otherwise, the complexity of ensuring compliance with performance-related constraints grows exponentially or super-exponentially as the degrees of freedom relating to the assignment of resources of data storage system grows. Among other reasons, this may be in large part to the sheer number of combinations of different assignments of many different resources. Moreover, reassigning resources from a current state to a combination that complies with some or all of the constraints, or does so more optimally than a current (or indeed any other) combination, may require significant effort on the part of the data storage system. This effort may overcome any benefit associated with the new combination and, accordingly, systems are required that determine new optimal arrangements wherein the costs of re-arrangement do not overcome the potential benefit of the more optimal arrangement.
As data centres increase in size, and complexity, an important observation that relates to data and data resources is that static resource allocation has become insufficient: (i) characteristics for a given set of data, including priority, change over time, (ii) data resources are, sometimes as a result of a change in data characteristics and sometimes as a result of other causes including failure, are also in rapid flux, and (iii) the constraints that guide resource allocation can change. As such, data placement that is limited to initial and/or static storage-related resource constraints is insufficient. Moreover, given the degree of complexity over data resource allocation for data sets, a manner of transitioning a given resource allocation into another resource allocation that does not overcome the benefit of an alternative data placement allocation is required.
This background information is provided to reveal information believed by the applicant to be of possible relevance. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art or forms part of the general common knowledge in the relevant art.